Modelling direct manipulation with Referent and Statecharts1

An approach to modelling mouse gestures in direct manipulation interfaces is presented. The Referent and Statechart languages are used for modelling structure and behaviour, respectively. A gesture is divided into a series of steps, driven by user action and aided by feedback, all modelled as Statechart states. The Referent describes dynamic relations between these states. The states and transitions determine the gesture syntax and the actual relations established and changed during recognition. We discuss how reasoning about these models can support the design of composite gesture recognisers and provide consistency checking and design critique.